JEP 471: Deprecate the Memory-Access Methods in sun.misc.Unsafe for Removal

AuthorRon Pressler & Alex Buckley
OwnerRon Pressler
TypeFeature
ScopeJDK
StatusClosed / Delivered
Release23
Componentcore-libs
Discussionjdk dash dev at openjdk dot org
Reviewed byAlan Bateman, Brian Goetz, Mark Reinhold, Maurizio Cimadamore, Paul Sandoz
Endorsed byAlan Bateman
Created2024/01/05 15:46
Updated2024/05/31 10:40
Issue8323072

Summary

Deprecate the memory-access methods in sun.misc.Unsafe for removal in a future release. These unsupported methods have been superseded by standard APIs, namely the VarHandle API (JEP 193, JDK 9) and the Foreign Function & Memory API (JEP 454, JDK 22). We strongly encourage library developers to migrate from sun.misc.Unsafe to supported replacements, so that applications can migrate smoothly to modern JDK releases.

Goals

Non-Goals

Motivation

The sun.misc.Unsafe class was introduced in 2002 as a way for Java classes in the JDK to perform low-level operations. Most of its methods — 79 out of 87 — are for accessing memory, either in the JVM's garbage-collected heap or in off-heap memory, which is not controlled by the JVM. As the name of the class suggests, these memory-access methods are unsafe: They can lead to undefined behavior, including JVM crashes. Therefore, they were not exposed as a standard API. They were neither envisaged for use by a broad range of clients nor intended to be permanent. Rather, they were introduced with the assumption that they were exclusively for use within the JDK, and that callers within the JDK would perform exhaustive safety checks before using them, and that safe standard APIs for this functionality would eventually be added to the Java Platform.

However, with no way in 2002 to prevent sun.misc.Unsafe from being used outside the JDK, its memory-access methods became a handy tool for library developers who wanted more power and performance than standard APIs could offer. For example, sun.misc.Unsafe::compareAndSwap can perform a CAS (compare-and-swap) operation on a field without the overhead of the java.util.concurrent.atomic API, while sun.misc.Unsafe::setMemory can manipulate off-heap memory without the 2GB limitation of java.nio.ByteBuffer. Libraries that do rely on ByteBuffer to manipulate off-heap memory, such as Apache Hadoop and Cassandra, use sun.misc.Unsafe::invokeCleaner to improve efficiency by deallocating off-heap memory promptly.

Unfortunately, not all libraries are diligent at performing safety checks before calling the memory-access methods, so there is a risk of failures and crashes in applications. Some uses of the methods are unnecessary, driven by the ease of copy-and-paste from online forums. Other uses of the methods may cause the JVM to disable optimizations, resulting in worse performance than if ordinary Java arrays had been used. Nevertheless, because use of the memory-access methods is so widespread, sun.misc.Unsafe was not encapsulated alongside other low-level APIs in JDK 9 (JEP 260). It remains available out-of-the-box in JDK 22, pending the availability of safe supported alternatives.

Over the past several years, we have introduced two standard APIs that are safe and performant replacements for the memory-access methods in sun.misc.Unsafe:

These standard APIs guarantee no undefined behavior, promise long-term stability, and have high-quality integration with the tooling and documentation of the Java Platform (examples of their use are given below). Given the availability of these APIs, it is now appropriate to deprecate and eventually remove the memory-access methods in sun.misc.Unsafe.

Removing the memory-access methods in sun.misc.Unsafe is part of a long-term coordinated effort to ensure that the Java Platform has integrity by default. Other initiatives include placing restrictions on the Java Native Interface (JNI, JEP 472) and on the dynamic loading of agents (JEP 451). These efforts will make the Java Platform more secure and more performant. They will also reduce the risk of application developers becoming trapped on older JDK releases due to libraries that break on newer releases when unsupported APIs are changed.

Description

The memory-access methods of sun.misc.Unsafe can be divided into three categories:

We will deprecate and remove the methods in phases, where each phase takes place in a separate JDK feature release:

  1. Deprecate all of the memory-access methods — on-heap, off-heap, and bimodal — for removal. This will cause compile-time deprecation warnings for code that refers to the methods, alerting library developers to their forthcoming removal. A new command-line option, described below, will enable application developers and users to receive runtime warnings when the methods are used.

    Distinct from deprecation warnings, javac has issued warnings about the use of sun.misc.Unsafe since 2006:

    warning: Unsafe is internal proprietary API and may be removed in a future release

    These warnings will continue to be issued, and cannot be suppressed.

  2. Issue a warning at run time, as detailed below, when a memory-access method is used, whether directly or via reflection. This will alert application developers and users to the forthcoming removal of the methods, and the need to upgrade libraries.

  3. Throw an exception when a memory-access method is used, whether directly or via reflection. This will further alert application developers and users to the imminent removal of the methods.

  4. Remove on-heap methods. These methods will be removed first because they have had standard replacements since JDK 9 in 2017.

  5. Remove off-heap and bimodal methods. These methods will be removed later because they have only had standard replacements since JDK 22 in 2023.

As to timing, we plan to implement

We may implement phases 4 and 5 simultaneously, if appropriate, when the time arrives.

Allowing use of the memory-access methods in sun.misc.Unsafe

The vast majority of Java developers do not use sun.misc.Unsafe explicitly in their own code. However, many applications depend, directly or indirectly, on libraries that use the memory-access methods of sun.misc.Unsafe. Starting in JDK 23, application developers can assess how the deprecation and removal of these methods will affect libraries by running with a new command line option, --sun-misc-unsafe-memory-access={allow|warn|debug|deny}. This option is similar, in spirit and in form, to the --illegal-access option introduced by JEP 261 in JDK 9. It works as follows:

An example of the warnings enabled by the option value warn is:

WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::setMemory has been called by com.foo.bar.Server (file:/tmp/foobarserver/thing.jar)
WARNING: Please consider reporting this to the maintainers of com.foo.bar.Server
WARNING: sun.misc.Unsafe::setMemory will be removed in a future release

The default value of the --sun-misc-unsafe-memory-access option will change from release to release as we proceed through the phases described above:

The following tools in the JDK can help advanced developers understand how their code uses deprecated methods in sun.misc.Unsafe:

sun.misc.Unsafe memory-access methods and their replacements

On-heap methods

These methods exist to obtain offsets or scales which are then used with the bimodal methods (below) to read and write fields or array elements. These use cases are now addressed by VarHandle and MemorySegment::ofArray.

In rare circumstances, these methods are used on their own to examine and manipulate the physical layouts of objects in memory (see examples here). There is no supported replacement for this use case; see below for further discussion.

The first three methods above were deprecated in JDK 18.

The fields associated with these methods are also deprecated for removal:

Off-heap methods

Bimodal memory-access methods

Migration examples

On-heap memory access

Suppose class Foo has an int field that we wish to double atomically. We can do that using sun.misc.Unsafe:

class Foo {

    private static final Unsafe UNSAFE = ...;    // A sun.misc.Unsafe object

    private static final long X_OFFSET;

    static {
        try {
            X_OFFSET = UNSAFE.objectFieldOffset(Foo.class.getDeclaredField("x"));
        } catch (Exception ex) { throw new AssertionError(ex); }
    }

    private int x;

    public boolean tryToDoubleAtomically() {
        int oldValue = x;
        return UNSAFE.compareAndSwapInt(this, X_OFFSET, oldValue, oldValue * 2);
    }

}

We can improve that to use the standard VarHandle API:

class Foo {

    private static final VarHandle X_VH;

    static {
        try {
            X_VH = MethodHandles.lookup().findVarHandle(Foo.class, "x", int.class);
        } catch (Exception ex) { throw new AssertionError(ex); }
    }

    private int x;

    public boolean tryAtomicallyDoubleX() {
        int oldValue = x;
        return X_VH.compareAndSet(this, oldValue, oldValue * 2);
    }

}

We can use sun.misc.Unsafe to perform a volatile write of an array element:

class Foo {

    private static final Unsafe UNSAFE = ...;

    private static final int ARRAY_BASE = UNSAFE.arrayBaseOffset(int[].class);
    private static final int ARRAY_SCALE = UNSAFE.arrayIndexScale(int[].class);

    private int[] a = new int[10];

    public void setVolatile(int index, int value) {
        if (index < 0 || index >= a.length)
            throw new ArrayIndexOutOfBoundsException(index);
        UNSAFE.putIntVolatile(a, ARRAY_BASE + ARRAY_SCALE * index, value);
    }

}

We can improve that to use VarHandle:

class Foo {

    private static final VarHandle AVH = MethodHandles.arrayElementVarHandle(int[].class);

    private int[] a = new int[10];

    public void setVolatile(int index, int value) {
        AVH.setVolatile(a, index, value);
    }

}

Off-heap memory access

Here is a class that uses sun.misc.Unsafe to allocate an off-heap buffer and perform three operations: a volatile write of an int, a bulk initialization of a subset of the buffer, and a copy of the buffer data into a Java int array:

class OffHeapIntBuffer {

    private static final Unsafe UNSAFE = ...;

    private static final int ARRAY_BASE = UNSAFE.arrayBaseOffset(int[].class);
    private static final int ARRAY_SCALE = UNSAFE.arrayIndexScale(int[].class);

    private final long size;
    private long bufferPtr;

    public OffHeapIntBuffer(long size) {
        this.size = size;
        this.bufferPtr = UNSAFE.allocateMemory(size * ARRAY_SCALE);
    }

    public void deallocate() {
        if (bufferPtr == 0) return;
        UNSAFE.freeMemory(bufferPtr);
        bufferPtr = 0;
    }

    private boolean checkBounds(long index) {
        if (index < 0 || index >= size)
            throw new IndexOutOfBoundsException(index);
        return true;
    }

    public void setVolatile(long index, int value) {
        checkBounds(index);
        UNSAFE.putIntVolatile(null, bufferPtr + ARRAY_SCALE * index, value);
    }

    public void initialize(long start, long n) {
        checkBounds(start);
        checkBounds(start + n-1);
        UNSAFE.setMemory(bufferPtr + start * ARRAY_SCALE, n * ARRAY_SCALE, 0);
    }

    public int[] copyToNewArray(long start, int n) {
        checkBounds(start);
        checkBounds(start + n-1);
        int[] a = new int[n];
        UNSAFE.copyMemory(null, bufferPtr + start * ARRAY_SCALE, a, ARRAY_BASE, n * ARRAY_SCALE);
        return a;
    }

}

We can improve that to use the standard Arena and MemorySegment APIs:

class OffHeapIntBuffer {

    private static final VarHandle ELEM_VH = ValueLayout.JAVA_INT.arrayElementVarHandle();

    private final Arena arena;
    private final MemorySegment buffer;

    public OffHeapIntBuffer(long size) {
        this.arena  = Arena.ofShared();
        this.buffer = arena.allocate(ValueLayout.JAVA_INT, size);
    }

    public void deallocate() {
        arena.close();
    }

    public void setVolatile(long index, int value) {
        ELEM_VH.setVolatile(buffer, 0L, index, value);
    }

    public void initialize(long start, long n) {
        buffer.asSlice(ValueLayout.JAVA_INT.byteSize() * start,
                       ValueLayout.JAVA_INT.byteSize() * n)
              .fill((byte) 0);
    }

    public int[] copyToNewArray(long start, int n) {
        return buffer.asSlice(ValueLayout.JAVA_INT.byteSize() * start,
                              ValueLayout.JAVA_INT.byteSize() * n)
                     .toArray(ValueLayout.JAVA_INT);
    }

}

Risks and Assumptions

Future Work

After deprecating the 79 memory-access methods for removal, sun.misc.Unsafe will contain only three methods that are not deprecated: